{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network with SPU\n",
    "\n",
    ">  Please read lab [Logistic Regression On SPU](./lr_with_spu.ipynb) first if you have not。\n",
    "\n",
    "In lab [Logistic Regression On SPU](./lr_with_spu.ipynb), we have showed how to use SecretFlow/SPU to convert a plaintext JAX training program to a secure MPC training program.\n",
    "\n",
    "In this lab, the idea is quite similar but this time we will work with a Neural Network model.\n",
    "\n",
    "We are going to use the same dataset and all the settings as lab [Logistic Regression On SPU](./lr_with_spu.ipynb).\n",
    "\n",
    "And first, let's work out the plaintext model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The following codes are demos only. It's **NOT for production** due to system security concerns, please **DO NOT** use it directly in production."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This tutorial needs more resources than 8c16g, which is the minimum requirement of SecretFlow."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a model with JAX/FLAX\n",
    "\n",
    "### Load the Dataset\n",
    "\n",
    "The below is just copied from lab [Logistic Regression On SPU](./lr_with_spu.ipynb). I'm not going to explain again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: flax==0.8.4 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (0.8.4)\n",
      "Requirement already satisfied: numpy==1.23.5 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (1.23.5)\n",
      "Requirement already satisfied: jax>=0.4.19 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (0.4.26)\n",
      "Requirement already satisfied: msgpack in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (1.1.0)\n",
      "Requirement already satisfied: optax in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (0.2.1)\n",
      "Requirement already satisfied: orbax-checkpoint in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (0.6.4)\n",
      "Requirement already satisfied: tensorstore in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (0.1.72)\n",
      "Requirement already satisfied: rich>=11.1 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (13.9.4)\n",
      "Requirement already satisfied: typing-extensions>=4.2 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (4.12.2)\n",
      "Requirement already satisfied: PyYAML>=5.4.1 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from flax==0.8.4) (6.0.2)\n",
      "Requirement already satisfied: ml-dtypes>=0.2.0 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from jax>=0.4.19->flax==0.8.4) (0.5.1)\n",
      "Requirement already satisfied: opt-einsum in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from jax>=0.4.19->flax==0.8.4) (3.4.0)\n",
      "Requirement already satisfied: scipy>=1.9 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from jax>=0.4.19->flax==0.8.4) (1.15.2)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from rich>=11.1->flax==0.8.4) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from rich>=11.1->flax==0.8.4) (2.19.1)\n",
      "Requirement already satisfied: absl-py>=0.7.1 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from optax->flax==0.8.4) (1.4.0)\n",
      "Requirement already satisfied: chex>=0.1.7 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from optax->flax==0.8.4) (0.1.7)\n",
      "Requirement already satisfied: jaxlib>=0.1.37 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from optax->flax==0.8.4) (0.4.26)\n",
      "Requirement already satisfied: etils[epath,epy] in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from orbax-checkpoint->flax==0.8.4) (1.12.2)\n",
      "Requirement already satisfied: nest_asyncio in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from orbax-checkpoint->flax==0.8.4) (1.6.0)\n",
      "Requirement already satisfied: protobuf in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from orbax-checkpoint->flax==0.8.4) (4.25.6)\n",
      "Requirement already satisfied: humanize in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from orbax-checkpoint->flax==0.8.4) (4.12.2)\n",
      "Requirement already satisfied: dm-tree>=0.1.5 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from chex>=0.1.7->optax->flax==0.8.4) (0.1.9)\n",
      "Requirement already satisfied: toolz>=0.9.0 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from chex>=0.1.7->optax->flax==0.8.4) (1.0.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=11.1->flax==0.8.4) (0.1.2)\n",
      "Requirement already satisfied: fsspec in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from etils[epath,epy]->orbax-checkpoint->flax==0.8.4) (2024.2.0)\n",
      "Requirement already satisfied: importlib_resources in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from etils[epath,epy]->orbax-checkpoint->flax==0.8.4) (6.5.2)\n",
      "Requirement already satisfied: zipp in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from etils[epath,epy]->orbax-checkpoint->flax==0.8.4) (3.21.0)\n",
      "Requirement already satisfied: attrs>=18.2.0 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from dm-tree>=0.1.5->chex>=0.1.7->optax->flax==0.8.4) (25.2.0)\n",
      "Requirement already satisfied: wrapt>=1.11.2 in /home1/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages (from dm-tree>=0.1.5->chex>=0.1.7->optax->flax==0.8.4) (1.14.1)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "# use new flax version for compatibility\n",
    "# fix numpy version to prevent install numpy2 and break other dependencies\n",
    "!{sys.executable} -m pip install flax==0.8.4 numpy==1.23.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "import jax\n",
    "\n",
    "\n",
    "def breast_cancer(party_id=None, train: bool = True) -> (np.ndarray, np.ndarray):\n",
    "    x, y = load_breast_cancer(return_X_y=True)\n",
    "    x = (x - np.min(x)) / (np.max(x) - np.min(x))\n",
    "    x_train, x_test, y_train, y_test = train_test_split(\n",
    "        x, y, test_size=0.2, random_state=42\n",
    "    )\n",
    "\n",
    "    if train:\n",
    "        if party_id:\n",
    "            if party_id == 1:\n",
    "                return x_train[:, :15], _\n",
    "            else:\n",
    "                return x_train[:, 15:], y_train\n",
    "        else:\n",
    "            return x_train, y_train\n",
    "    else:\n",
    "        return x_test, y_test"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Model\n",
    "\n",
    "\n",
    "We are going to use a 4-layer [MLP](https://en.wikipedia.org/wiki/Multilayer_perceptron) model with a [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) activation function here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Sequence\n",
    "import flax.linen as nn\n",
    "\n",
    "\n",
    "FEATURES = [30, 15, 8, 1]\n",
    "\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    features: Sequence[int]\n",
    "\n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        for feat in self.features[:-1]:\n",
    "            x = nn.relu(nn.Dense(feat)(x))\n",
    "        x = nn.Dense(self.features[-1])(x)\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define the training method here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "\n",
    "\n",
    "def predict(params, x):\n",
    "    # TODO(junfeng): investigate why need to have a duplicated definition in notebook,\n",
    "    # which is not the case in a normal python program.\n",
    "    from typing import Sequence\n",
    "    import flax.linen as nn\n",
    "\n",
    "    FEATURES = [30, 15, 8, 1]\n",
    "\n",
    "    class MLP(nn.Module):\n",
    "        features: Sequence[int]\n",
    "\n",
    "        @nn.compact\n",
    "        def __call__(self, x):\n",
    "            for feat in self.features[:-1]:\n",
    "                x = nn.relu(nn.Dense(feat)(x))\n",
    "            x = nn.Dense(self.features[-1])(x)\n",
    "            return x\n",
    "\n",
    "    return MLP(FEATURES).apply(params, x)\n",
    "\n",
    "\n",
    "def loss_func(params, x, y):\n",
    "    pred = predict(params, x)\n",
    "\n",
    "    def mse(y, pred):\n",
    "        def squared_error(y, y_pred):\n",
    "            return jnp.multiply(y - y_pred, y - y_pred) / 2.0\n",
    "\n",
    "        return jnp.mean(squared_error(y, pred))\n",
    "\n",
    "    return mse(y, pred)\n",
    "\n",
    "\n",
    "def train_auto_grad(x1, x2, y, params, n_batch=10, n_epochs=10, step_size=0.01):\n",
    "    x = jnp.concatenate((x1, x2), axis=1)\n",
    "    xs = jnp.array_split(x, len(x) / n_batch, axis=0)\n",
    "    ys = jnp.array_split(y, len(y) / n_batch, axis=0)\n",
    "\n",
    "    def body_fun(_, loop_carry):\n",
    "        params = loop_carry\n",
    "        for x, y in zip(xs, ys):\n",
    "            _, grads = jax.value_and_grad(loss_func)(params, x, y)\n",
    "            params = jax.tree_util.tree_map(\n",
    "                lambda p, g: p - step_size * g, params, grads\n",
    "            )\n",
    "        return params\n",
    "\n",
    "    params = jax.lax.fori_loop(0, n_epochs, body_fun, params)\n",
    "    return params\n",
    "\n",
    "\n",
    "def model_init(n_batch=10):\n",
    "    model = MLP(FEATURES)\n",
    "    return model.init(jax.random.PRNGKey(1), jnp.ones((n_batch, FEATURES[0])))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the Model\n",
    "\n",
    "We use AUC as the validation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "\n",
    "def validate_model(params, X_test, y_test):\n",
    "    y_pred = predict(params, X_test)\n",
    "    return roc_auc_score(y_test, y_pred)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BUILD Together\n",
    "\n",
    "Let's put everything together and train a plaintext NN model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc=0.9927939731411726\n"
     ]
    }
   ],
   "source": [
    "# Load the data\n",
    "x1, _ = breast_cancer(party_id=1, train=True)\n",
    "x2, y = breast_cancer(party_id=2, train=True)\n",
    "\n",
    "\n",
    "# Hyperparameter\n",
    "n_batch = 10\n",
    "n_epochs = 10\n",
    "step_size = 0.01\n",
    "\n",
    "\n",
    "# Train the model\n",
    "init_params = model_init(n_batch)\n",
    "params = train_auto_grad(x1, x2, y, init_params, n_batch, n_epochs, step_size)\n",
    "\n",
    "# Test the model\n",
    "X_test, y_test = breast_cancer(train=False)\n",
    "auc = validate_model(params, X_test, y_test)\n",
    "print(f'auc={auc}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Must keep the number of AUC in mind, we are going to repeat the training with SPU. Let's do that magic!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Train a Model with SPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The version of SecretFlow: 1.11.0b1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "2025-03-25 16:22:04,154\tINFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.\n",
      "2025-03-25 16:22:04,276\tINFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.\n",
      "/home/zoupeicheng.zpc/miniconda3/envs/sf/lib/python3.10/subprocess.py:1796: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.\n",
      "  self.pid = _posixsubprocess.fork_exec(\n",
      "2025-03-25 16:22:05,905\tINFO worker.py:1724 -- Started a local Ray instance.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m(pyu_fn pid=58922)\u001b[0m [2025-03-25 16:22:08.052] [warning] [openssl_factory.cc:83] Yacl has been configured to use Yacl's entropy source, but unable to find one. Fallback to use openssl's default entropy srouce\n",
      "\u001b[36m(pyu_fn pid=58922)\u001b[0m [2025-03-25 16:22:08.053] [warning] [openssl_factory.cc:83] Yacl has been configured to use Yacl's entropy source, but unable to find one. Fallback to use openssl's default entropy srouce\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[36m(pyu_fn pid=58923)\u001b[0m 2025-03-25 16:22:08,017,017 INFO [xla_bridge.py:backends:863] Unable to initialize backend 'cuda': \n",
      "\u001b[36m(pyu_fn pid=58923)\u001b[0m 2025-03-25 16:22:08,017,017 INFO [xla_bridge.py:backends:863] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'\n",
      "\u001b[36m(pyu_fn pid=58923)\u001b[0m 2025-03-25 16:22:08,017,017 INFO [xla_bridge.py:backends:863] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "import secretflow as sf\n",
    "\n",
    "# Check the version of your SecretFlow\n",
    "print('The version of SecretFlow: {}'.format(sf.__version__))\n",
    "\n",
    "# In case you have a running secretflow runtime already.\n",
    "# in new version of sf.shutdown() will return Error if no runtime is found.\n",
    "try:\n",
    "    sf.shutdown()\n",
    "except:\n",
    "    pass\n",
    "\n",
    "sf.init(['alice', 'bob'], address='local')\n",
    "\n",
    "alice, bob = sf.PYU('alice'), sf.PYU('bob')\n",
    "spu = sf.SPU(sf.utils.testing.cluster_def(['alice', 'bob']))\n",
    "\n",
    "x1, _ = alice(breast_cancer)(party_id=1, train=True)\n",
    "x2, y = bob(breast_cancer)(party_id=2, train=True)\n",
    "init_params = model_init(n_batch)\n",
    "\n",
    "\n",
    "device = spu\n",
    "x1_, x2_, y_ = x1.to(device), x2.to(device), y.to(device)\n",
    "init_params_ = sf.to(alice, init_params).to(device)\n",
    "\n",
    "params_spu = spu(train_auto_grad, static_argnames=['n_batch', 'n_epochs', 'step_size'])(\n",
    "    x1_, x2_, y_, init_params_, n_batch=n_batch, n_epochs=n_epochs, step_size=step_size\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check params from SPU program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'params': {'Dense_0': {'bias': array([ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, -8.4447712e-03,\n",
      "        4.7277778e-02,  3.7619472e-04,  0.0000000e+00,  4.5648813e-03,\n",
      "        0.0000000e+00, -3.4031734e-02, -8.4131360e-03,  0.0000000e+00,\n",
      "        0.0000000e+00,  5.6682736e-02, -4.8433840e-03,  0.0000000e+00,\n",
      "        3.5732210e-02,  6.3549578e-03,  2.9711574e-03,  3.2665402e-02,\n",
      "        0.0000000e+00, -2.1323502e-02, -7.8181922e-03,  0.0000000e+00,\n",
      "        2.8501630e-02,  0.0000000e+00, -3.0902624e-03,  3.8683414e-05,\n",
      "        1.4437497e-02,  2.0847648e-02], dtype=float32), 'kernel': array([[-0.14871399, -0.23531966, -0.1493772 , -0.01558919, -0.13323164,\n",
      "         0.19175903, -0.03680335, -0.03745084, -0.14176767,  0.03231028,\n",
      "         0.12652716, -0.40251398, -0.16895528,  0.21399274, -0.13845327,\n",
      "         0.10585146, -0.11602777,  0.38624388,  0.05965905,  0.06317489,\n",
      "         0.07793002, -0.01319648, -0.28804976, -0.09602834,  0.111113  ,\n",
      "        -0.08544238,  0.07546462, -0.04119562, -0.3826748 ,  0.23766656],\n",
      "       [ 0.17794432,  0.22939444, -0.24440876, -0.148503  ,  0.33701146,\n",
      "         0.02583458, -0.04215176,  0.41052234,  0.32438686, -0.16436441,\n",
      "         0.08169016,  0.05258645,  0.31134155,  0.29317677,  0.1226961 ,\n",
      "        -0.38753128, -0.3853491 , -0.06536682, -0.25915533, -0.33227926,\n",
      "        -0.31588346, -0.29118308, -0.06018971,  0.22978456,  0.10113718,\n",
      "        -0.0131004 ,  0.17880836, -0.23216496,  0.3828004 ,  0.03805932],\n",
      "       [ 0.14288743, -0.02135937,  0.16818418,  0.08982229, -0.38853168,\n",
      "        -0.04851243,  0.13870293, -0.05801402,  0.2847225 , -0.12711686,\n",
      "         0.25702122,  0.09647745,  0.11669752, -0.18962513, -0.03995326,\n",
      "        -0.09573793,  0.07307927,  0.14649987,  0.09225577,  0.03891373,\n",
      "        -0.24625295,  0.0372512 , -0.01914966, -0.2521044 ,  0.17078328,\n",
      "         0.24981512, -0.00287431, -0.09845665,  0.20796731,  0.08842857],\n",
      "       [-0.22512461, -0.00447209, -0.04558258, -0.04286569, -0.1305536 ,\n",
      "        -0.34271502, -0.00925776,  0.0901569 , -0.28249556,  0.2202184 ,\n",
      "         0.11645912,  0.0447506 , -0.05022046,  0.29517627, -0.23622917,\n",
      "         0.05994216,  0.2596401 ,  0.18784146,  0.14600964,  0.29655722,\n",
      "         0.03958386,  0.16070746, -0.11334923, -0.06969583,  0.2647735 ,\n",
      "        -0.31787586,  0.08121131,  0.2556237 , -0.05900668,  0.1953014 ],\n",
      "       [-0.24255335,  0.0796814 , -0.06769416, -0.11747566,  0.18755525,\n",
      "        -0.0613867 , -0.05366787, -0.06935148,  0.07923843,  0.02540354,\n",
      "        -0.31858033,  0.28704095, -0.06028181, -0.3014956 , -0.17661634,\n",
      "         0.07973169,  0.16141297,  0.3279425 ,  0.20514369,  0.30348164,\n",
      "         0.27109915,  0.27654904,  0.07070905,  0.20799829, -0.07334277,\n",
      "        -0.10325597,  0.01552792,  0.31757554,  0.31676418, -0.06809753],\n",
      "       [ 0.07325247,  0.06063734,  0.05307055,  0.17843898,  0.18786696,\n",
      "         0.17703715,  0.08110298,  0.01481734, -0.04425612,  0.06165443,\n",
      "         0.2882646 ,  0.05877423,  0.2642675 ,  0.12086734, -0.0218261 ,\n",
      "        -0.15167004, -0.04630694,  0.00737368,  0.28398234,  0.1007943 ,\n",
      "        -0.3035403 , -0.31351358, -0.17609796,  0.11222892,  0.11561264,\n",
      "        -0.27606148, -0.06868617,  0.06135447,  0.30819774, -0.28001517],\n",
      "       [-0.25858742, -0.01557118,  0.27712947, -0.38401195,  0.39872247,\n",
      "        -0.12920067, -0.02737655,  0.1757228 ,  0.13031279,  0.15869968,\n",
      "        -0.02626206,  0.2941114 ,  0.03558348,  0.03587052, -0.29661202,\n",
      "        -0.1697014 ,  0.02999383,  0.16187142, -0.17532967, -0.0876811 ,\n",
      "        -0.04855376, -0.10537766,  0.08300757, -0.04356985, -0.25447172,\n",
      "        -0.09856978, -0.041673  , -0.04678066, -0.3353476 , -0.11826624],\n",
      "       [ 0.27912214, -0.07000907, -0.02482197,  0.04388492, -0.08831042,\n",
      "        -0.00139722,  0.08730513, -0.2483547 ,  0.15355726, -0.12888038,\n",
      "        -0.00434992, -0.00280654, -0.0779365 , -0.10295209,  0.24092257,\n",
      "        -0.2569998 ,  0.2917946 , -0.1947996 , -0.27556112,  0.01552372,\n",
      "         0.12702632,  0.12880231,  0.15365958,  0.14312764,  0.06206781,\n",
      "        -0.11138317,  0.05906248, -0.11649914,  0.01586545, -0.20324321],\n",
      "       [ 0.06792215,  0.08562456, -0.09105198,  0.17886123,  0.07519971,\n",
      "        -0.13828582,  0.33566758, -0.14806081, -0.03185603,  0.39236438,\n",
      "        -0.13354063,  0.19828129,  0.05120739, -0.04608051, -0.12948763,\n",
      "        -0.22251473,  0.12676883,  0.39127854, -0.11602704,  0.00092509,\n",
      "        -0.07845783,  0.17063616,  0.27078646,  0.06742911,  0.07425453,\n",
      "        -0.00924775, -0.00354199, -0.36856014, -0.12303323,  0.22055578],\n",
      "       [ 0.02833255, -0.12450694,  0.17980424,  0.15363526,  0.05482823,\n",
      "         0.19171023, -0.09493516,  0.06867197, -0.07678868, -0.01939397,\n",
      "         0.05700703, -0.39339092,  0.05287276,  0.3794906 ,  0.2464098 ,\n",
      "        -0.12122658,  0.03999835, -0.38035634,  0.19540724, -0.09051453,\n",
      "         0.32060203,  0.01484731, -0.03493971,  0.11108357, -0.33724397,\n",
      "        -0.30602393, -0.11325404,  0.159679  ,  0.06750803,  0.10088514],\n",
      "       [ 0.16804892,  0.19497408, -0.09764463, -0.14558761,  0.10151543,\n",
      "        -0.31742722, -0.11584353, -0.2865643 , -0.10121611, -0.1301302 ,\n",
      "         0.19798903, -0.06929785,  0.00182401, -0.06140092, -0.23813443,\n",
      "         0.14183137,  0.4120589 , -0.11150961, -0.07696664, -0.03938428,\n",
      "        -0.05823901, -0.25617248,  0.17550963, -0.05777389,  0.04626928,\n",
      "         0.12045561,  0.31443763, -0.18237962, -0.16254546, -0.09767339],\n",
      "       [-0.06191421, -0.1155844 ,  0.0726438 ,  0.12529258,  0.20683448,\n",
      "         0.15766354, -0.08057442, -0.19450316,  0.02132484,  0.2354292 ,\n",
      "        -0.17701548, -0.3511722 , -0.22017696,  0.03137182,  0.10099782,\n",
      "        -0.4008683 , -0.13381556, -0.0659401 , -0.14123085, -0.17201565,\n",
      "        -0.06661722,  0.09940286, -0.03092401,  0.25938374,  0.06447111,\n",
      "        -0.25064036, -0.03492744,  0.08023162,  0.25556132, -0.24089926],\n",
      "       [ 0.0089751 , -0.40074265, -0.0630265 ,  0.06182706,  0.3735702 ,\n",
      "         0.03176737,  0.27502176, -0.2881164 , -0.20248234,  0.1611321 ,\n",
      "        -0.21795315,  0.10631423,  0.00266191, -0.2730174 ,  0.07528849,\n",
      "         0.07777505,  0.02632871,  0.09457071, -0.28338334,  0.02558231,\n",
      "         0.17132387,  0.047729  , -0.01300138, -0.09193201,  0.22021295,\n",
      "        -0.19897453,  0.34152788,  0.08680119, -0.08853401, -0.00905181],\n",
      "       [ 0.12034325,  0.12541285, -0.36260542,  0.22371292, -0.07335918,\n",
      "         0.10497788,  0.00435905, -0.08325318,  0.2286281 , -0.14954633,\n",
      "         0.08158794, -0.3141489 ,  0.08761807, -0.05525999,  0.08568221,\n",
      "        -0.02317639, -0.22309203,  0.0285781 ,  0.1041774 ,  0.0975873 ,\n",
      "         0.08703597,  0.0155434 ,  0.1736699 ,  0.08375281, -0.01751347,\n",
      "        -0.0653795 ,  0.05047977, -0.2294519 ,  0.05721854,  0.25090066],\n",
      "       [-0.39711425,  0.10012019, -0.07080775, -0.16265565, -0.13911583,\n",
      "         0.16160382,  0.1602145 , -0.00789458,  0.05427746,  0.16592917,\n",
      "         0.22369803,  0.36696285,  0.06149234, -0.04858233,  0.3345179 ,\n",
      "        -0.07261233,  0.193892  , -0.06003518, -0.30302724,  0.17182067,\n",
      "         0.29064044,  0.21396407,  0.04790886,  0.09809829,  0.10329911,\n",
      "         0.12731801, -0.05798505,  0.15246147, -0.36663866,  0.17798509],\n",
      "       [ 0.01063947,  0.08888602,  0.2685741 ,  0.3439614 ,  0.06943247,\n",
      "         0.30761197, -0.15886995,  0.00264697,  0.04297216, -0.06384344,\n",
      "        -0.01198623, -0.10141458,  0.0390074 , -0.02127311,  0.13492535,\n",
      "        -0.16071658, -0.27638686, -0.11029266,  0.12214167, -0.25607044,\n",
      "        -0.08863828,  0.03597008, -0.1732463 ,  0.12558362,  0.1478779 ,\n",
      "         0.0970176 ,  0.17842568,  0.08070078,  0.0718725 ,  0.08295526],\n",
      "       [ 0.14691211,  0.13539682, -0.05013725, -0.2566475 , -0.23767036,\n",
      "         0.21671404,  0.13727273, -0.03883488,  0.39051494,  0.00476654,\n",
      "         0.14543669, -0.06966858, -0.15188436,  0.06678236, -0.24012773,\n",
      "         0.3115953 ,  0.0662728 , -0.2530469 , -0.20176558, -0.22605042,\n",
      "         0.13813484, -0.14101902,  0.3429038 ,  0.1295524 ,  0.28457743,\n",
      "         0.06187631, -0.22961026,  0.2912134 , -0.08083458, -0.34454456],\n",
      "       [-0.01825674,  0.12697385,  0.11828473, -0.08935875, -0.04363643,\n",
      "        -0.06176041, -0.11145921, -0.06697068, -0.34101105, -0.2551305 ,\n",
      "        -0.14840548, -0.2012809 , -0.00368209,  0.05239139,  0.06488028,\n",
      "         0.08271401,  0.05891104,  0.21343401, -0.13793978,  0.30933118,\n",
      "        -0.09877008, -0.15072922, -0.10378313,  0.03409073,  0.09370108,\n",
      "        -0.22453101,  0.35971865,  0.2400895 , -0.03083891, -0.10381848],\n",
      "       [-0.14539114,  0.17940336,  0.01638718, -0.2706321 , -0.02601299,\n",
      "        -0.03973655, -0.03252296,  0.03152584, -0.15473387, -0.09656349,\n",
      "         0.04075833,  0.12999703, -0.19559053, -0.17638862,  0.12239662,\n",
      "        -0.26904342,  0.27144247, -0.0700524 , -0.07336612,  0.03824541,\n",
      "         0.22631611,  0.30120707,  0.02216698, -0.30002955, -0.06067093,\n",
      "        -0.07689847, -0.37136936,  0.19664557, -0.10526311, -0.2740894 ],\n",
      "       [ 0.05383714,  0.03157908, -0.00410651, -0.04451686, -0.10077155,\n",
      "        -0.06426768,  0.31361273, -0.13607043,  0.12432167, -0.10924786,\n",
      "        -0.03941241,  0.22020288, -0.07174791,  0.08708778,  0.04954614,\n",
      "         0.36316663,  0.00659108, -0.1583945 ,  0.09209342, -0.17415541,\n",
      "        -0.14203098,  0.38341957,  0.02246684,  0.00735356, -0.02806289,\n",
      "        -0.15888669,  0.03910065, -0.09437944,  0.21786483,  0.01439759],\n",
      "       [-0.093013  , -0.19803196, -0.31412685,  0.17170613,  0.133173  ,\n",
      "        -0.14113761, -0.21318686, -0.16238129,  0.0943417 ,  0.14660117,\n",
      "         0.01858073, -0.0221183 , -0.14671399,  0.39277712, -0.20137532,\n",
      "         0.10903868, -0.02885686, -0.15210135,  0.1743122 ,  0.07787172,\n",
      "         0.09585002,  0.102861  ,  0.18952502, -0.15745284,  0.09723206,\n",
      "        -0.26545548, -0.05130725,  0.0804038 ,  0.05854797,  0.24786004],\n",
      "       [ 0.05508049,  0.23070964,  0.00277765, -0.05163932, -0.13319278,\n",
      "         0.17230488, -0.03837845,  0.17233688, -0.12099533, -0.12201309,\n",
      "        -0.16572389, -0.08696222, -0.01523115, -0.3166936 ,  0.19612928,\n",
      "        -0.20850554,  0.34564465,  0.252585  ,  0.03058532, -0.2363572 ,\n",
      "        -0.02455695, -0.07402438,  0.18045619,  0.0800463 ,  0.02340357,\n",
      "         0.05160186,  0.08303359, -0.10962112,  0.20512374,  0.05485076],\n",
      "       [-0.29294893,  0.0158273 , -0.00053273,  0.07538971,  0.17627099,\n",
      "         0.16074777,  0.0059112 , -0.02527514, -0.27194142, -0.26422063,\n",
      "         0.17577434,  0.26843715, -0.16067582,  0.00550777, -0.4134903 ,\n",
      "         0.13214995,  0.20718713, -0.09203249, -0.321208  ,  0.03001113,\n",
      "        -0.03515691, -0.11420843,  0.0069131 ,  0.06026545,  0.31073356,\n",
      "         0.31373236,  0.15468088,  0.238434  ,  0.20546623,  0.16574574],\n",
      "       [-0.13176121, -0.09717393, -0.21108817,  0.30688956,  0.13686776,\n",
      "         0.2546361 , -0.2318604 ,  0.26539138, -0.20205683,  0.26761878,\n",
      "        -0.01473916, -0.17002106, -0.00166056,  0.1011616 ,  0.28316772,\n",
      "        -0.10187809,  0.02499999,  0.09213223, -0.06187937,  0.05129597,\n",
      "        -0.13957006,  0.2983255 ,  0.16424753, -0.20014518,  0.10159925,\n",
      "        -0.09227319, -0.09285475, -0.24736921,  0.2820006 ,  0.18463379],\n",
      "       [-0.19597173, -0.26224336, -0.02397527,  0.14056443, -0.05118129,\n",
      "         0.09831372,  0.10008648,  0.0876383 , -0.20916207, -0.04817775,\n",
      "         0.11512281, -0.01074605,  0.06285878, -0.14395382,  0.18310134,\n",
      "         0.18600374, -0.01793548, -0.01051368,  0.29881978,  0.02923457,\n",
      "         0.15022175, -0.02574207,  0.10514745,  0.32682592, -0.06475702,\n",
      "        -0.07946987, -0.33096194, -0.33395487,  0.14654058, -0.18609725],\n",
      "       [-0.04333222,  0.18819539,  0.03159693,  0.11939715, -0.22679462,\n",
      "         0.09432116, -0.08721486,  0.25599957, -0.14890689,  0.09946157,\n",
      "         0.187719  , -0.19527301,  0.08275311, -0.14670566, -0.12542172,\n",
      "        -0.13777602,  0.09573571, -0.2980532 ,  0.10540853, -0.11812344,\n",
      "        -0.23555467, -0.01769879, -0.2976231 ,  0.04321702, -0.04170205,\n",
      "         0.04330476, -0.10866146,  0.35263634,  0.27451855,  0.01663768],\n",
      "       [ 0.17763095, -0.07081565, -0.1255958 , -0.13399567, -0.22848447,\n",
      "        -0.20404285,  0.07889004,  0.13384144, -0.31692368, -0.13477243,\n",
      "        -0.08197729,  0.02778099,  0.02475752,  0.10588112, -0.25831395,\n",
      "        -0.24312648,  0.0376173 ,  0.05451217, -0.13535275, -0.10998507,\n",
      "        -0.31393322,  0.0512618 , -0.00060911, -0.15892605, -0.17078651,\n",
      "         0.23628204,  0.0846691 ,  0.01051676,  0.08872144,  0.16417605],\n",
      "       [ 0.37543845,  0.06814782,  0.07721338, -0.40397   , -0.05512369,\n",
      "         0.00878023,  0.33256996,  0.18473433, -0.07991335,  0.20011076,\n",
      "         0.14145647, -0.15846947, -0.1596188 , -0.18773371, -0.17598417,\n",
      "        -0.13405162,  0.2131348 , -0.1309074 ,  0.10694368,  0.28709382,\n",
      "         0.13358122,  0.3303783 , -0.26874876, -0.22376873,  0.293559  ,\n",
      "        -0.0348874 ,  0.14831385,  0.12624305, -0.20834129,  0.05822696],\n",
      "       [ 0.1786205 , -0.12086539, -0.07798681,  0.16461018,  0.13113365,\n",
      "         0.11193126, -0.02917075, -0.01835178,  0.03707664, -0.39162362,\n",
      "         0.04380277,  0.12685025, -0.2031177 ,  0.14990912,  0.08968332,\n",
      "        -0.14305945,  0.37688786,  0.25453225,  0.09407996,  0.30029792,\n",
      "         0.00201203, -0.03300798, -0.3196806 ,  0.08428498, -0.1035914 ,\n",
      "         0.15461805,  0.15203737, -0.00354651, -0.15649001,  0.03190117],\n",
      "       [-0.33499128, -0.18705532,  0.12659647,  0.27141514, -0.04179814,\n",
      "        -0.01660369, -0.15886804,  0.14642526,  0.10316479,  0.13912395,\n",
      "         0.26202357,  0.03827994,  0.17041308,  0.281385  ,  0.03019579,\n",
      "        -0.21716599,  0.05987941,  0.20940788,  0.27819836, -0.30284634,\n",
      "         0.21740744,  0.06876232, -0.01624343, -0.09320645,  0.1671552 ,\n",
      "        -0.05673492, -0.01679376, -0.33967906,  0.04148224,  0.24174266]],\n",
      "      dtype=float32)}, 'Dense_1': {'bias': array([-0.00928336, -0.00438911,  0.06738339,  0.        , -0.01120922,\n",
      "        0.01074508, -0.02256708, -0.00346468,  0.03708997,  0.09008977,\n",
      "        0.        ,  0.        ,  0.        , -0.01539148,  0.03887193],\n",
      "      dtype=float32), 'kernel': array([[-0.21578984, -0.08009043, -0.34168613, -0.0361702 , -0.04044062,\n",
      "        -0.1927943 ,  0.07815595,  0.38473648, -0.2709835 ,  0.03089066,\n",
      "        -0.11207438,  0.12150717,  0.3848398 ,  0.12946524,  0.03025971],\n",
      "       [ 0.30358258,  0.14899506,  0.02244115,  0.17263442,  0.00111011,\n",
      "        -0.16067751,  0.17209713, -0.19851246, -0.00883473,  0.06376015,\n",
      "        -0.09706706, -0.27143684,  0.3290201 ,  0.01247436, -0.20563015],\n",
      "       [ 0.01422057,  0.25236645,  0.26592228, -0.07877432,  0.02570081,\n",
      "         0.13746086, -0.30379143, -0.3006696 ,  0.22853018,  0.07396485,\n",
      "        -0.05445625,  0.06825568, -0.1147591 , -0.04364532, -0.00258724],\n",
      "       [-0.03808632,  0.36381817, -0.05992284,  0.16604961, -0.18014544,\n",
      "         0.17623937, -0.2444128 , -0.31743556, -0.06891848, -0.04920346,\n",
      "         0.13664721, -0.0523686 ,  0.12887284,  0.2582376 , -0.06481002],\n",
      "       [ 0.01581764, -0.0054702 ,  0.06451058,  0.00376759,  0.05298941,\n",
      "         0.09622377, -0.3335601 ,  0.15771125, -0.0031673 ,  0.34260106,\n",
      "        -0.0192093 , -0.01579043,  0.10246339, -0.02411184,  0.14005014],\n",
      "       [-0.15425196, -0.1891739 , -0.12516877, -0.15351656, -0.20896591,\n",
      "        -0.03577305,  0.01807149,  0.1685001 ,  0.05936436,  0.0377558 ,\n",
      "         0.07395943,  0.03353617,  0.06905574,  0.15163349, -0.26086116],\n",
      "       [ 0.09053175,  0.31335747, -0.17576236,  0.05338556, -0.19664493,\n",
      "         0.22920157,  0.21464722,  0.1493361 ,  0.30394968, -0.24031784,\n",
      "         0.11673151, -0.04494652, -0.03595138,  0.30892205, -0.01469775],\n",
      "       [-0.09351237, -0.0924224 ,  0.29310796,  0.13807979,  0.14410564,\n",
      "         0.11154778,  0.1920185 , -0.22070102,  0.00914793, -0.00838362,\n",
      "        -0.10840161, -0.04924423,  0.15356652,  0.38943473, -0.15210427],\n",
      "       [ 0.00290038,  0.18365669,  0.03764611, -0.01739281,  0.18317279,\n",
      "         0.00409579,  0.09654756,  0.07968089,  0.21979392,  0.22736616,\n",
      "        -0.15136842,  0.20434377,  0.11873652, -0.33702517,  0.11251152],\n",
      "       [-0.0370011 ,  0.05358464, -0.0042589 , -0.00428124, -0.2019563 ,\n",
      "        -0.12829909,  0.0629352 ,  0.13846506, -0.17897698, -0.38954097,\n",
      "        -0.07185909,  0.22984825, -0.11224893,  0.04145162, -0.38177186],\n",
      "       [ 0.23527995,  0.16633692, -0.08347486,  0.2034623 , -0.20409596,\n",
      "        -0.07193303,  0.11208355,  0.24517196,  0.23958978, -0.13913625,\n",
      "        -0.02639589, -0.11256775, -0.2708755 , -0.00493087,  0.130058  ],\n",
      "       [-0.0557088 , -0.3465374 ,  0.29848823, -0.16680802,  0.06142375,\n",
      "         0.09287459,  0.14722499, -0.12598759, -0.01329686, -0.26824528,\n",
      "         0.08740366,  0.10008687,  0.126474  ,  0.13801362,  0.25637314],\n",
      "       [ 0.01379359, -0.19647817,  0.1487906 ,  0.03884296, -0.14404036,\n",
      "         0.35002947, -0.03261705, -0.11960487, -0.3504191 , -0.09014206,\n",
      "         0.16814655, -0.17364143, -0.26453283,  0.18936165, -0.30342686],\n",
      "       [ 0.15264304, -0.16593818,  0.28034884, -0.02613993,  0.09317499,\n",
      "        -0.1145373 , -0.02915497,  0.09113975,  0.16308393,  0.16566753,\n",
      "        -0.16354224, -0.0239345 ,  0.21729939, -0.37558755,  0.36439762],\n",
      "       [ 0.2754505 , -0.05118339,  0.03051634,  0.38373795,  0.18914255,\n",
      "        -0.30550474, -0.13651967, -0.09851488, -0.08356322, -0.17306487,\n",
      "         0.00162853,  0.2703498 , -0.01431669,  0.01418361, -0.23040965],\n",
      "       [-0.11281794, -0.08905558,  0.05266505, -0.03345683,  0.1795441 ,\n",
      "         0.15272224, -0.05195545,  0.10905784,  0.2167249 , -0.05777529,\n",
      "         0.29314327, -0.27227783,  0.2271789 , -0.0416695 ,  0.08242023],\n",
      "       [ 0.11221188,  0.15371764, -0.13823096, -0.18225348, -0.26140004,\n",
      "         0.22891754, -0.12165031, -0.20521307,  0.3913149 ,  0.19771382,\n",
      "         0.00469573, -0.04090644, -0.17770383,  0.22471263,  0.24130823],\n",
      "       [-0.08917232, -0.133329  ,  0.11583245, -0.31593603, -0.05461437,\n",
      "        -0.03294215,  0.17573108, -0.03389841, -0.04563008, -0.00729044,\n",
      "        -0.20087469, -0.04283047, -0.06482439,  0.0017387 ,  0.0841424 ],\n",
      "       [ 0.10726382,  0.15352634,  0.0963406 ,  0.01950403, -0.00730222,\n",
      "        -0.25290012, -0.23461862,  0.35619122,  0.17616239, -0.18047178,\n",
      "        -0.25238752, -0.05561563, -0.20358025, -0.13480246,  0.144216  ],\n",
      "       [-0.32129222,  0.01505694,  0.32079297,  0.30846062,  0.06560643,\n",
      "        -0.20671539,  0.07110591,  0.0910629 , -0.05796371,  0.06883919,\n",
      "        -0.24341525,  0.09922644, -0.39771098, -0.1435731 ,  0.18188469],\n",
      "       [ 0.17828378, -0.3734979 , -0.34489787, -0.18513836, -0.12530549,\n",
      "        -0.3593558 , -0.21523722,  0.40664414, -0.06089024, -0.12822454,\n",
      "         0.30890357, -0.05408919,  0.1326272 ,  0.01791927,  0.22214782],\n",
      "       [-0.1597906 , -0.19275309, -0.39843786, -0.13795093,  0.2481104 ,\n",
      "        -0.30260313,  0.25339955,  0.36626542, -0.0446806 ,  0.20680596,\n",
      "         0.10090984, -0.17185582, -0.0115933 ,  0.28760862,  0.07138807],\n",
      "       [-0.38753942, -0.21487829, -0.3494261 , -0.3746059 ,  0.00249292,\n",
      "        -0.3801347 , -0.2602242 ,  0.06026101, -0.0513204 ,  0.24081683,\n",
      "         0.20540601, -0.09037866, -0.16683164,  0.24142444, -0.26693586],\n",
      "       [-0.20973277, -0.0101587 ,  0.16557135,  0.20874728, -0.19013675,\n",
      "        -0.31780738, -0.03113298, -0.06459363,  0.39771876, -0.26641545,\n",
      "         0.3113753 , -0.06382816, -0.39697587,  0.10766909,  0.01153809],\n",
      "       [ 0.18971755,  0.02601397,  0.10644442, -0.21744527, -0.26413256,\n",
      "         0.15006709,  0.13827504, -0.21841131, -0.06611386,  0.27945745,\n",
      "        -0.10811286, -0.32185346, -0.03605992,  0.04212931, -0.01747657],\n",
      "       [ 0.04841596, -0.1742564 ,  0.12262946,  0.32726845, -0.08306259,\n",
      "        -0.3148705 ,  0.10644473,  0.09955123,  0.07176274, -0.20581172,\n",
      "         0.04141927, -0.00283223,  0.15970391,  0.19535124, -0.21868774],\n",
      "       [ 0.18083718,  0.09281985, -0.27907234, -0.32184833,  0.08461116,\n",
      "        -0.13168144, -0.22216935,  0.06936353,  0.1084511 , -0.15438795,\n",
      "        -0.02530034,  0.03963596, -0.01773681,  0.04080668,  0.15702324],\n",
      "       [ 0.1722859 , -0.27421778,  0.03915332, -0.10644263,  0.15347561,\n",
      "        -0.40776026, -0.14519481, -0.1972003 ,  0.1516371 ,  0.08711421,\n",
      "        -0.01810022,  0.03163131, -0.31662256, -0.08890387, -0.31581816],\n",
      "       [-0.09769952, -0.02877441,  0.35823053, -0.27105692,  0.32774943,\n",
      "         0.08072305,  0.30246264, -0.19247323, -0.17831784,  0.29237252,\n",
      "         0.09354727, -0.25247368,  0.12927192,  0.38658634, -0.39495987],\n",
      "       [ 0.06277542,  0.08469228, -0.00951533,  0.10955815,  0.09936444,\n",
      "        -0.19083323,  0.21161583,  0.3930413 ,  0.0044056 ,  0.20088935,\n",
      "        -0.13770127,  0.27256176, -0.09585688, -0.05922067,  0.33084267]],\n",
      "      dtype=float32)}, 'Dense_2': {'bias': array([-0.01853457,  0.10750265,  0.        , -0.02408628,  0.10219114,\n",
      "        0.11779837, -0.00741374,  0.        ], dtype=float32), 'kernel': array([[ 1.70725286e-01, -9.90791768e-02, -2.29398310e-02,\n",
      "         1.22194603e-01, -1.06561959e-01, -3.67063880e-02,\n",
      "         1.41975164e-01,  8.84004533e-02],\n",
      "       [ 1.80475801e-01,  1.18611023e-01,  5.27172148e-01,\n",
      "        -1.53576493e-01, -3.86221558e-02, -1.22964129e-01,\n",
      "         6.26538694e-03,  1.59997195e-02],\n",
      "       [ 1.44997954e-01,  5.34001708e-01, -3.44355702e-01,\n",
      "        -9.43375081e-02,  1.35723338e-01, -8.21426511e-03,\n",
      "         1.19161576e-01, -4.11049873e-01],\n",
      "       [ 2.49562308e-01,  2.14895070e-01,  3.24747413e-01,\n",
      "        -4.91989821e-01, -1.14358604e-01, -2.11471766e-02,\n",
      "         7.69384056e-02,  3.31377953e-01],\n",
      "       [ 4.12398577e-02, -1.21448338e-01, -3.10940951e-01,\n",
      "         3.62135768e-01, -2.74279505e-01, -5.16959608e-01,\n",
      "        -5.41965663e-02,  5.59358776e-01],\n",
      "       [-4.59978878e-02, -6.31723851e-02,  1.12806559e-01,\n",
      "         3.72395128e-01,  1.19805902e-01,  2.30253339e-01,\n",
      "        -1.38900071e-01,  4.47024256e-02],\n",
      "       [-6.63796067e-02, -3.46704423e-01, -4.84587103e-01,\n",
      "         1.17089301e-01, -2.02461258e-01, -3.72340798e-01,\n",
      "         5.67471743e-01, -2.42814660e-01],\n",
      "       [ 3.36931825e-01, -1.21137604e-01,  3.77372414e-01,\n",
      "         4.15773451e-01,  4.28525954e-02, -3.28820586e-01,\n",
      "         4.96313989e-01, -1.31856978e-01],\n",
      "       [ 3.29436958e-01,  7.28745013e-02,  2.03800440e-01,\n",
      "         1.12697646e-01,  3.64428461e-01,  1.61250755e-01,\n",
      "        -2.09109038e-01,  5.50374389e-04],\n",
      "       [ 1.63456440e-01,  4.27246094e-04, -4.44933087e-01,\n",
      "        -9.06846225e-02,  3.35955620e-01,  4.89923239e-01,\n",
      "        -1.45878404e-01, -2.39348367e-01],\n",
      "       [-2.70905137e-01,  1.98488906e-01,  1.63834617e-01,\n",
      "        -3.97947073e-01,  8.93427432e-02,  3.55304182e-01,\n",
      "        -4.33428586e-03, -2.55934328e-01],\n",
      "       [ 2.71844625e-01, -6.50432259e-02,  5.75310588e-02,\n",
      "         8.60967785e-02,  4.62493598e-02,  6.84029907e-02,\n",
      "        -3.49441171e-01,  3.20650488e-01],\n",
      "       [ 3.87197286e-01,  1.02545783e-01, -3.67730916e-01,\n",
      "        -1.37638271e-01, -2.77009606e-03, -9.75389779e-03,\n",
      "        -2.03590691e-02, -3.78600478e-01],\n",
      "       [ 1.50329724e-01, -1.60095647e-01, -4.73224610e-01,\n",
      "         2.41277456e-01,  4.34344560e-02, -3.39222312e-01,\n",
      "         1.99609771e-01, -1.64232552e-01],\n",
      "       [-9.00831074e-02,  1.48896769e-01, -3.05355638e-02,\n",
      "         4.30267960e-01, -3.87655109e-01,  5.50715029e-01,\n",
      "        -9.88794565e-02, -3.92895550e-01]], dtype=float32)}, 'Dense_3': {'bias': array([0.23491548], dtype=float32), 'kernel': array([[-0.07592535],\n",
      "       [ 0.6634375 ],\n",
      "       [ 0.19396022],\n",
      "       [-0.21857159],\n",
      "       [ 0.44743913],\n",
      "       [ 0.72588986],\n",
      "       [-0.07620324],\n",
      "       [-0.20487784]], dtype=float32)}}}\n"
     ]
    }
   ],
   "source": [
    "params_spu = spu(train_auto_grad)(x1_, x2_, y_, init_params)\n",
    "params = sf.reveal(params_spu)\n",
    "print(params)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's validate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc=0.9927939731411726\n"
     ]
    }
   ],
   "source": [
    "X_test, y_test = breast_cancer(train=False)\n",
    "auc = validate_model(params, X_test, y_test)\n",
    "print(f'auc={auc}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the end of the lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sf",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
